Introduction

Every day, important scientific findings are rejected at large. From man-made climate change to the safety and efficacy of Covid-19 vaccinations, science skepticism has run rampant among lay consumers in modern society (Hornsey & Fielding, 2017). To increase public faith in science, some have proposed the use of crowd science (Silberzahn et al., 2018; Uhlmann et al., 2019).
\(~~~~~\) We explore the effects of scientific findings emerging from a crowd of researchers (vs. a typical research collaboration) on lay perceptions of scientific findings. In line with social norm theory (Miller & Prentice, 2016), we expect that observing consensus among a crowd (the consistent crowd condition) will – compared to the conclusion of a single scientist (the single estimate condition) – increase conformity in opinion. Drawing from work on intuitive statistics (Gigerenzer & Murray, 2015), we also expect laypeople to intuitively accord to the logic of the wisdom of crowds: the ability of an aggregate of multiple estimates (rather than a single estimate) to reduce noise stemming from individual bias or error (Schweinsberg et al., 2021).
\(~~~~~\) In contrast, when crowd estimates show low consensus and high variance (the inconsistent crowd condition), we predict that observers will be less swayed and more likely to attribute the findings to bias and error. In addition, due to the difficulty of lay reasoning about variation (Ben-Zvi & Garfield, 1999), we predict an aversion to variability: i.e., we expect that observing variable estimates will decrease lay confidence in the precise average parameter estimate in both crowd conditions.

Hypotheses

Table 1: Predicted differences with the single estimate condition

Note. We regress each outcome on prior beliefs and condition (with the single estimate condition as the reference category). When laypeople observe multiple consistent (inconsistent) estimates from a crowd, we expect – compared to a single estimate and controlling for prior beliefs – higher (lower) posterior beliefs and credibility of the results, lower confidence in the precise average parameter estimate, and lower (higher) ratings of bias and error.

Open Science: Preregistration, survey, data, and code available at
\(~~\) github.com/shilaan/many-analysts
\(~~\) osf.io/vedb4

Methods

We ran an experiment (N = 1,498; UK/US Prolific) with three conditions
\(~\) Single estimate
A single parameter estimate (5%)
\(~\) Consistent crowd
Multiple crowd estimates: low variance, high consensus (M = 5%)
\(~\) Inconsistent crowd
Multiple crowd estimates: high variance, low consensus (M = 5%)

Experimental Design

Results

Figure 1: Estimates of differences with the single estimate condition In line with our hypotheses, lay consumers of inconsistent crowd estimates (vs. a single estimate)…
\(~~\) Have lower posterior beliefs about the reported phenomenon
\(~~\) Find the results less credible
\(~~\) Have less confidence in the average estimate of 5%
\(~~\) Are more likely to attribute the average estimate (5%) to bias
\(~~\) Are more likely to attribute the average estimate (5%) to error

Contrary to our hypotheses, lay consumers of consistent crowd estimates (vs. a single estimate)…
\(~~\) Have lower posterior beliefs about the reported phenomenon
\(~~\) Are more likely to attribute the average estimate (5%) to error

We found no significant effects for lay consumers of consistent crowd estimates (vs. a single estimate) on…
\(~~\) Credibility of the results
\(~~\) Confidence in the average estimate
\(~~\) Ratings of bias

Exploratory results

For the additional exploratory measure, lay consumers of consistent and inconsistent crowd estimates…
\(~~\) Perceive greater discretion (i.e., idiosyncratic choices)

Figure 2: Distribution of prior and posterior beliefs by condition

In terms of belief updating, Figure 2 shows a positive difference within the consistent crowd condition \((\)pre vs. post \(M_d\) = 4.75 [2.55,6.95]\()\), but less so than for the single estimate condition \((M_d\) = 11.66 [9.66,13.66]\()\). As expected, we find negative belief updating in the inconsistent crowd condition \((M_d\) = -11.45 [-13.75,-9.16]\()\).

Conclusion

\(~~\) Compared to providing a single estimate, we find no evidence that
\(~~~~~~~\) crowd estimates improve lay perceptions of scientific findings

Future directions
\(~~\) Does variability aversion explain the findings?
\(~~~\) Perceptions of scientists
\(~~\) Science communication and communicating uncertainty